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Abstract 
Assessing youth psychopathology involves collecting multiple informants’ reports. Yet, multi- 
informant reports often disagree, necessitating integrative strategies that optimize predictive 
power. The Trait score approach leverages principal components analysis (PCA) to account for 
the context and perspective from which informants provide reports. This approach may boost the 
predictive power of multi-informant reports and thus warrants rigorous testing. We tested the 
Trait score approach using multi-informant reports of adolescent social anxiety in a mixed 
clinical/community sample of adolescents (n=127). The Trait score incrementally predicted 
observed social anxiety (fs: .47-.67) and referral status (ORs: 2.66-6.53), above-and-beyond 
individual informants’ reports and a composite of informants’ reports. The Trait score predicted 
observed behavior at magnitudes well above those typically observed for individual informants’ 
reports of internalizing psychopathology (i.e., rs=.01-.15). Findings demonstrate the ability of 
the Trait score to improve prediction of clinical indices, and potentially transform widely used 


practices in multi-informant assessments. 
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Integrating Multiple Informants’ Reports: How Conceptual and Measurement Models May 
Address Long-Standing Problems in Clinical Decision-Making 
Imagine you are a clinician working with an adolescent who reports a consistent and 
impairing fear of social situations, including an intense fear of negative evaluation from others. 
Behaviorally, the adolescent avoids social situations and in particular those that involve 
interacting with unfamiliar individuals. When unable to avoid these situations, the adolescent 
reports enduring them with great distress. In contrast, the adolescent’s parent reports minimal 
levels of social anxiety and instead identifies behaviors displayed at home that result in the 
adolescent under-performing at school (e.g., poor schoolwork, rule-breaking) as important to 
focus on in treatment. As a clinician, how will you use this information to inform the care that 
you provide? How will you conceptualize the adolescent’s social anxiety, and what diagnoses 
will you assign? Is one informant “right” and the other “wrong”? As you begin treatment 
planning, which adolescent problems ought to be targeted first? 
Suboptimal Use of Multi-Informant Reports: A Clinical Decision-Making Problem 
In this paper, we address commonly encountered questions in clinical practice by 

advancing an innovative strategy for integrating multi-informant reports of youth mental health. 
Indeed, we opened this paper with a case example that highlights a common clinical observation: 
Youth vary considerably in the contexts in which they display mental health symptoms (e.g., 
home, school; Achenbach, McConaughy, & Howell, 1987) and significant others in their lives 
(e.g., parents, teachers) vary in their capacities for observing them within and across these 
contexts (Dirks, De Los Reyes, Briggs-Gowan, Cella, & Wakschlag, 2012). To account for these 
complexities in clinical presentation, clinicians rely on multiple informants to provide reports of 


youth mental health (Hunsley & Mash, 2007). Yet, when implementing this multi-informant 
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approach, informants’ reports display overall low-to-moderate levels of convergence (i.e., 7s 

= .20-.30s; Achenbach et al., 1987; De Los Reyes et al., 2015). Consequently, how a mental 
health professional interprets these informant discrepancies has profound implications for crucial 
clinical decision-making tasks including estimating prognosis, planning treatment, and 
evaluating treatment effectiveness (Hawley & Weisz, 2003). Yet, despite decades of research 
attention on these topics, there is little evidence that supports the integrated use of multi- 
informant reports (De Los Reyes & Langer, 2018; Hunsley & Mash, 2007). Does integrating 
multi-informant data improve the accuracy, outcome, or efficiency of clinical work, relative to 
alternative practices like choosing a single, “optimal” informant, or examining each informant’s 
report individually? Without an answer to this question, the field lacks clear guidelines for 
clinicians and researchers seeking to leverage multi-informant reports to improve clinical 
decision-making (Beidas et al., 2015; Marsh, De Los Reyes, & Lilienfeld, 2018). 

In practice, low correspondence often results in assessors shunning multi-informant 
approaches to assessment and instead making clinical decisions that rely on data from a single 
source (Loeber, Green, & Lahey, 1990; Loeber, Green, Lahey, & Stouthamer-Loeber, 1989; 
Marsh, Zeveney, & De Los Reyes, 2019). This practice likely reflects long-held assumptions that 
low informant correspondence signals measurement error or informant biases (De Los Reyes, 
2011). Needless to say, the evidence suggests that no single informant’s report robustly predicts 
clinical indices germane to service delivery. For example, any one informant’s report relates at 
only small-to-moderate levels with observed youth behavior, including for externalizing 
problems (e.g., 7s = .02-.52; Becker, Luebbe, Fite, Stoppelbein, & Greening, 2014; Doctoroff & 
Arnold, 2004; Henry & Metropolitan Area Child Study Research Group, 2006; Hinshaw, Han, 


Erhardt, & Huber, 1992; Wakschlag et al., 2008; Winsler & Wallace, 2002) and internalizing 
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problems (e.g., 7s = .01-.33; Becker et al., 2014; Beidel, Turner, Hamlin, & Morris, 2000a; 
Cartwright-Hatton, Hodges, & Porter, 2003; Hinshaw et al., 1992; Winsler & Wallace, 2002). In 
fact, correlations between informants’ reports and youth behavior vary widely, with most 
correlations falling in the small range (1.e., majority of vs = .01-.15). Crucially, practitioners and 
researchers in youth mental health lack instruments that consistently—and at large 
magnitudes—predict key clinical indices relevant to youth mental health service delivery and 
research. We need efficient yet comprehensive assessments that, when integrated, improve 
decision-making in clinical and research settings (Beidas et al., 2015). 
Addressing the Clinical Decision-Making Problem: The Trait Score Approach 

Nearly 20 years ago, Kraemer and colleagues (2003) proposed an integrative approach 
with potential for addressing these longstanding assessment issues. We refer to this strategy as 
the “Trait score approach” and it reflects an insightful observation by Kraemer and colleagues: 
To arrive at a clinically useful set of multi-informant reports, one must select informants who 
systematically vary in the contexts and perspectives from which they observe youth. Stated 
differently, assessors should select informants who, for systematic reasons, tend to provide 
discrepant reports. This reflects an innovative departure from historical thinking on low 
convergence in multi-informant assessments. Rather than considering low informant 
convergence as a problem for decision-making, assessors might leverage low convergence to 
enhance decision-making. 

The underlying assumptions of the Trait score approach fundamentally differ from 
assumptions underlying alternative strategies for integrating continuous, multivariate data. For 
instance, researchers often take a Composite score approach to integrate multi-informant reports, 


essentially by taking the simple average among continuous scores taken from informants’ reports 
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(De Los Reyes, Alfano, Lau, Augenstein, & Borelli, 2016). The Composite score approach has 
its reference to the very origins of classical test theory (Borsboom, 2005). In particular, 
Edgeworth (1888) asserted that variations among individual raters of the same behavior reflect 
“error” around a “true score” representation of the behavior being assessed. In this respect, the 
Composite score approach treats variations among informants’ reports as error. In contrast, the 
Trait score approach assumes that reports vary from one another both systematically and due to 
measurable factors. 

Kraemer and colleagues (2003) advanced a mathematical model for sources of variability 
that contribute to an informant’s rating of youth symptoms: 

Informant's Rating = Trait + Context + Perspective 

Kraemer and colleagues’ model harkens back to decades of research on techniques for 
aggregating multivariate data (e.g., factor analysis; Tabachnick & Fidell, 2001). In fact, Kraemer 
and colleagues’ innovation is in leveraging a data aggregation technique that has historically 
been used to parsimoniously aggregate multiple item responses on a measure of someone’s 
behavior to, instead, parsimoniously aggregate multiple informants’ ratings on a measure of 
someone’s behavior. To implement their theoretical approach, Kraemer and colleagues applied 
principal components analysis (PCA) to a carefully curated set of multi-informant reports. This 
statistical approach is consistent with 50 years of research that points to robustly low levels of 
correspondence among informants’ reports (Achenbach et al., 1987; De Los Reyes et al., 2015). 
Further, mental health professionals commonly solicit the reports of informants who qualitatively 
vary in the contexts in which they observe the behaviors about which they provide reports (e.g., 
parents at home vs. teachers at school) as well as the perspectives from which they provide 


reports (e.g., self-reports vs. observer-reports; De Los Reyes et al., 2013). The PCA approach 


INTEGRATING MULTIPLE INFORMANTS’ REPORTS 7 


advanced by Kraemer and colleagues (2013) requires selecting informants who vary across these 
systematic factors and thus assumes that the factors explaining variability among the informants’ 
reports entered into the model are linear combinations of orthogonal variables. This approach 
reflects a conservative and falsifiable strategy for integrating multi-informant data. Indeed, to the 
degree that the data entered into the model violate assumptions of orthogonality among 
informants’ reports, the resulting information gleaned from the model should fail to predict 
clinically relevant criterion variables, over-and-above alternative approaches. 

The Kraemer and colleagues (2003) approach involves synthesizing the variability among 
informants’ reports into three sources: Context (i.e., environment in which the informant 
observes youth), Perspective (i.e., whether the report comes from an observer- or self-rater), and 
Trait (1.e., concerns that manifest across contexts and perspectives). This approach is designed to 
identify these three sources of variability among informants’ reports through extraction of three 
components, with specific sources of variability being identified through examination of 
component weights. That is, Kraemer and colleagues posit that when one selects informants who 
vary in their contexts and perspectives, one component (i.e., Trait score) reflects variability for 
which all three informants’ reports load strongly and in the same direction. Further, two 
additional components reflect variability for which informants’ reports load consistent with a 
Context score (1.e., informants from different contexts load in opposite directions) and 
Perspective score (i.e., self-reports load in the opposite direction of observer informants’ 
reports). By isolating a component reflecting concerns that manifest across contexts and 
perspectives, Kraemer and colleagues proposed that the Trait score optimizes multi-informant 
reports for maximal utility in predicting a wide array of clinical outcomes. This approach, which 


holds great promise, has yet to be rigorously tested. 
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Building an Evidence Base for Selecting Informants: The Operations Triad Model 

To optimize the predictive power of the approach, Kraemer and colleagues (2003) argue 
that one should use a “mix-and-match” criterion to identify those informants who systematically 
vary in sources of variability (Figure 1). This approach is similar to Global Positioning System 
(GPS) technologies that use data from three satellites to triangulate on a precise location in 
space. With GPS, one cannot achieve triangulation with data from a single satellite, nor can one 
arrive at triangulated data using three satellites situated in the same location in space. Similarly, 
no single “optimal informant” exists to assess youth mental health, and three similar informants 
(e.g., three teachers) would not provide effective triangulation given their position in the same 
space or environment (e.g., school). Importantly, the process by which one identifies triangulated 
satellites requires data to ensure that the satellites are, in fact, situated in optimal locations in 
space. Similarly, the 7rait score approach holds value insofar as an evidence base exists to 
support that low correspondence among reports reflects systematic processes. The Operations 
Triad Model (OTM; De Los Reyes et al., 2013) provides a guide for building such an evidence 
base for multi-informant assessments. In particular, the OTM facilitates detecting circumstances 
in which low convergence among multi-informant reports reflects meaningful, systematic 
information about clinical phenomena. As such, the OTM facilitates determining which 
informants might be most appropriate to use when applying the 7rait score approach. 

Several studies informed by the OTM reveal that systematic patterns of multi-informant 
reports facilitate characterizing clinical presentations. For example, relative to patterns of parent- 
teacher reports that diverge in estimates of youth autism spectrum disorder symptoms, patterns of 
parent-teacher reports that converge on moderate or high levels of such symptoms point to 


increased clinical severity of the youth being evaluated (e.g., medication, diagnosis; Lerner, De 
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Los Reyes, Drabick, Gerber, & Gadow, 2017). Interestingly, these patterns in multi-informant 
reports also have predictive utility: In a recent study of psychiatric inpatient intake assessments, 
patterns of agreement or disagreement in parent-adolescent reports of adolescent affective 
symptoms facilitated prediction of treatment outcomes (Makol, De Los Reyes, Ostrander, & 
Reynolds, 2019). In fact, an emerging body of evidence indicates that informants who vary in 
their contexts and perspectives provide meaningful information about cross-contextual 
consistencies in behavior (De Los Reyes, Henry, Tolan, & Wakschlag, 2009; Hartley et al., 
2011; Kwon et al., 2012), mental health service use (Jones et al., 2019; Makol & Polo, 2018), 
and treatment outcomes (Becker-Haimes, Jensen-Doss, Birmaher, Kendall, & Ginsburg, 2017). 
In sum, research across psychopathology domains, informants, measurement methods, and 
developmental periods support that multi-informant reporting patterns yield meaningful clinical 
information. This work supports a key assumption of Kraemer and colleagues’ (2003) Trait score 
approach: Variations among informants’ reports reflect systematic factors germane to the clinical 
presentations of those assessed. In turn, informants should be selected for the unique information 
they provide, in an effort to optimize the utility of youth mental health assessments. 
The Need for Rigorous, Conceptually Grounded Research 

Despite the promise of Kraemer and colleagues’ (2003) approach, it is historically 
underutilized and has yet to be rigorously examined. This is largely due to the lack of conceptual 
frameworks and basic science to guide researchers in selecting appropriate informants to 
leverage in assessments. Further, although Kraemer and colleagues propose that the Trait score 
is the primary predictive tool derived from their approach, it remains an empirical question as to 
whether the Context and Perspective scores also yield valuable clinical information. Beyond 


these conceptual issues, practical challenges have curtailed rigorous tests of Trait scores. Given 
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resource limitations, assessments rarely include the three informants necessary to implement the 
approach. This occurs despite the fact that evidence-based assessment of youth mental health 
entails collecting multi-informant reports (Hunsley & Mash, 2007). Even among studies that 
estimate Trait scores, use of non-parallel measures across informants hinder the interpretability 
of study findings (e.g., Perren, Stadelmann, Burgin, & Von Klitzing, 2006). Further, prior Trait 
score studies have been largely limited to use of parent, youth, and teacher reports (Perren et al., 
2006; Noordhof, Oldehinkel, Verhulst, & Ormel, 2008) and examinations of youth externalizing 
problems (Houts, Caspi, Pianta, Arseneault, & Moffit, 2010). Finally, no study to our knowledge 
has examined whether the 7rait score displays incremental validity in predicting independent 
criterion variables (e.g., observed behavior, clinical status), over-and-above alternative strategies 
for either examining informants’ reports individually or aggregating data (e.g., composite score). 
Current Study 

Using Kraemer and colleagues’ (2003) approach, we surmised that a Trait score derived 
from a set of strategically selected informants will significantly boost the power of multi- 
informant assessments to predict criterion variables taken using other modalities, over-and-above 
the levels of correspondence typically observed between informants’ reports and these modalities 
(i.e., 7s = .01-.15; Becker et al., 2014; Beidel et al., 2000a; Cartwright-Hatton et al., 2003; 
Hinshaw et al., 1992; Winsler & Wallace, 2002). In this study, we applied the 7rait score 
approach to social anxiety assessments in a mixed clinical/community sample of adolescents. To 
provide a conservative test of this approach, we examined whether the Trait score predicted 
independent criterion variables above-and-beyond well-established measures of social anxiety 
taken from individual informants’ reports. We addressed three aims. First, we expected the social 


anxiety Trait score to demonstrate incremental validity in predicting observed adolescent social 
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anxiety in controlled laboratory tasks, relative to well-established measures of social anxiety 


taken from individual informants’ reports. Second, we expected the social anxiety Trait score to 


demonstrate incremental validity in distinguishing adolescents on referral status (1.e., clinic- 
referred vs. community control), relative to well-established measures of social anxiety taken 
from individual informants’ reports. Third, we expected the social anxiety Trait score to provide 
incremental prediction of criterion variables, over-and-above an alternative strategy for 
integrating multi-informant reports, namely the Composite score approach. 
Research Demonstration: Applying the 7rait Score Approach to Adolescent Social Anxiety 
In line with the example we presented at the opening of this paper, we demonstrate the 
utility of the Trait score approach with an innovative, evidence-based approach to multi- 
informant assessments of adolescent social anxiety. We seek to use this demonstration to 
highlight how the Trait score approach addresses two key limitations in evidence-based 
assessment of youth mental health problems broadly, and adolescent social anxiety in particular. 
First, when parents and adolescents disagree, assessors often make judgments that more closely 
align with parent reports, relative to adolescent reports (Brown-Jacobsen, Wallace, & Whiteside, 
2011; Hawley & Weisz, 2003). In social anxiety assessments, parent-adolescent discrepancies 
often result in the interpretation that a social desirability bias leads adolescents to downplay or 
underreport their social anxiety symptoms (De Los Reyes et al., 2015). In contrast, by placing an 
emphasis on selecting informants who systematically vary in the context and perspective from 
which they rate youth mental health symptoms, the Trait score approach forces discrepancies 
among reports. That is, by normalizing the presence of discrepant reports, the approach provides 


methods for integrating these reports. 
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Second, parents serve as “gatekeepers” who play a crucial role in initiating mental health 
care and thus evidence-based assessments involve collecting both parent and adolescent reports, 
particularly for internalizing concerns (Makol & Polo, 2018). Yet, prior work indicates that 
relying only on these two informants yields insufficient clinical information. First, relative to 
children from earlier developmental periods, parents of adolescents have reduced access to and 
understanding of their adolescent’s activities and relationships with peers (Smetana, 2008). 
Adolescent social anxiety often manifests in interactions with peers, making the peer context a 
key target of treatment (Hofmann et al., 1999). In support of these ideas, parent reports of 
adolescent social anxiety fail to predict adolescents’ self-reported experiences in interactions 
with unfamiliar peers (Deros et al., 2018). Parents’ lack of access to the peer context likely 
contributes to the low correspondence often observed between parent and adolescent reports of 
adolescent social anxiety. In contrast, to triangulate on the Trait score, the Kraemer and 
colleagues (2003) approach requires three carefully selected informants who vary in context and 
perspective. 

When assessing adolescent social anxiety, which informants should be leveraged in the 
Trait score approach? In their self-reports, adolescents provide information from a self- 
perspective and across contexts. In contrast, parents provide information about mental health 
concerns observed in the home context, but often do not directly observe adolescents in non- 
home contexts. Although teachers commonly stand in as a third informant in multi-informant 
assessments of youth externalizing problems, they may not be the key third informant to leverage 
when applying the Trait score approach to adolescent social anxiety. Given that information 
about the unfamiliar peer context is crucially lacking in assessments of adolescent social anxiety, 


an optimal informant would have opportunities to observe the adolescent within social 
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interactions with unfamiliar peers. Commonly used methods for collecting peers’ reports, such as 
peer nominations, provide information that is predictive of youth social anxiety (Gazelle, 
Workman, & Allan, 2010). However, these approaches often require large groups of peers to 
provide valid estimates of functioning for any one adolescent, and lack feasibility due to ethical 
concerns (e.g., obtaining parent consent to collect peer reports; Card & Hodges, 2008). 

Recent work in evidence-based assessment of adolescent social anxiety supports use of a 
third informant: unfamiliar peer confederates. These individuals “stand in” as unfamiliar peers 
and interact with adolescents in brief, structured social interactions designed to elicit anxiety 
(Anderson & Hope, 2009; Beidel, Rao, Scharfstein, Wong, & Alfano, 2010; Deros et al., 2018). 
In fact, unfamiliar peer confederate reports yield psychometrically sound data that (a) hold 
incremental value in predicting adolescent social anxiety concerns relative to parent reports, (b) 
distinguish adolescents on referral status, and (c) predict adolescents’ anxiety-related experiences 
in social interactions as indexed by both adolescent self-repots of state arousal and observer 
ratings (Deros et al., 2018; Glenn et al., 2019). Taken together, unfamiliar peer confederates 
provide rich information about adolescents’ non-home, peer contexts, and thus represent an ideal 
third informant to report on adolescent social anxiety. In the present study, we leverage a 
curated, evidence-based set of informants that consist of parents, adolescents, and unfamiliar 
peer confederates, to assess adolescent social anxiety using the Trait score approach. 

Method 
Participants 

We recruited study participants from the Washington, DC, Maryland, and Northern 

Virginia areas using advertisements posted online (e.g., Craigslist) and flyers posted in local 


businesses (e.g., cafes, libraries, doctor’s offices). Participants responded to one of two posted 
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advertisements: (1) study providing a no-cost social anxiety evaluation for adolescents (i.e., 
clinic-referred adolescents) and (2) nonclinical study on family relationships (1.e., community 
control adolescents). All study procedures described below were held consistent across 
participants, regardless of referral status. 

Eligible dyads were required to: (a) be fluent in English, (b) understand the consenting 
and interview process, and (c) have an adolescent aged 14-15 years who the parent did not report 
as having a history of learning or developmental disabilities. The total sample included 127 
caregiver-adolescent dyads (43 clinic-referred, 84 community control). Adolescents were 14 or 
15 years old (Mage = 14.46, SD = 0.50) and 85 adolescents were female (66.9%). Based on parent 
report, adolescents’ race/ethnicity included (values exceed 100% given that multiple categories 
could be selected): African American/Black (53.5%), Caucasian/European American/White 
(33.1%), Hispanic/Spanish/Latino/a (10.2%), Asian American/Asian (5.5%), American Indian 
(0.8%), other (7.9%). Caregivers included the adolescent’s biological mother/father (95.3%) or 
other caregiver (4.7%). Caregivers reported household income using a scale with 10 categories 
that varied by $100 increments. Based on this scale, 26.0% of families had a weekly household 
income of $500 or less, 22.0% had a weekly household income between $501 and $900, and 
52.0% had a weekly household income of $901 or more per week. 

In all study analyses, we pooled the clinic-referred and community control samples. This 
approach mimics displays of dimensionally varying social anxiety in the general population and 
is consistent with dimensional models of psychopathology (Casey, Oliveri, & Insel, 2014). 
Further, we selected this approach given prior work indicating that dimensional approaches to 
assessing psychopathology exhibit greater reliability and validity relative to categorical 


approaches (Markon, Chmielewski, & Miller, 2011). Nonetheless, we conducted analyses to 
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determine whether clinic-referred and community control groups differed on key demographic 
characteristics (i.e., adolescent age/gender, adolescent racial/ethnic background, family income, 
parent’s relationship to adolescent, parent’s marital status). Given the exploratory nature of these 
tests, we applied a Bonferroni correction (i.e., 11 tests and thus a corrected p-value of .0045) and 
observed non-significant differences between the clinic-referred and community control samples, 
thus justifying our pooled sample approach. 
Procedure 

We received Institutional Review Board approval for all study procedures. Prior to 
beginning the study, caregivers provided consent for themselves and their adolescent. 
Adolescents and caregivers completed a battery of counterbalanced survey measures using 
Qualtrics software. After completing survey measures, adolescents participated in 
counterbalanced mock social interaction tasks with trained peer confederates. We trained 
undergraduate research assistants to interact as unfamiliar peer confederates with adolescent 
participants across mock social interaction tasks. Upon completion of the study, families 
received $100 in monetary compensation for participation. 
Measures 

Survey measures. Parents completed a demographic questionnaire. Parents, adolescents, 
and unfamiliar peer confederates completed a battery of survey measures to assess the 
adolescent’s social anxiety. These informants completed measures with parallel survey items, 
with only minor modification made to fit each informant’s perspective (i.e., “My child” for 
caregiver measures, “I” for adolescent measures, “The participant” for unfamiliar peer 
confederate measures). As described below, unfamiliar peer confederates completed survey 


measures following their interactions with adolescents in social interaction tasks. 
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Our aims involved testing the validity of the Trait score approach including the 
incremental validity of this approach above-and-beyond alternative approaches (i.e., individual 
reports and Composite score approach). This necessitated use of three different measures of 
adolescent social anxiety, so as to avoid method confounds in measures used across the various 
approaches. First, parents, adolescents, and unfamiliar peer confederates completed the Social 
Interaction Anxiety Scale (SIAS; Mattick & Clarke, 1998), a 20-item measure for assessing 
social anxiety concerns that adolescents may experience during social interactions (example 
item: “I have difficulty making eye contact with others.”). Informants rated items on a scale 
ranging from 0 (Not at all characteristic or true of me) to 4 (Extremely characteristic or true of 
me). Informants’ reports on the SIAS display convergent validity and distinguish adolescents on 
referral status (Deros et al., 2018; Glenn et al., 2019). 

Parents and adolescents also completed the widely used Social Phobia and Anxiety 
Inventory for Children (SPAIC; Beidel, Turner, & Morris, 1995), a 26-item measure in which 
informants endorse how often the adolescent feels nervous or scared when in various social 
scenarios (e.g., public speaking). Several items include “sub-items” that require informants to 
rate the adolescent’s social anxiety with different interaction partners (e.g., “adults” vs. “boys or 
girls his/her age that he/she knows” vs. “boys or girls his/her age that he/she doesn’t know”). 
These sub-items were averaged at the item level to create a composite score. Parents and 
adolescents rated items using a scale ranging from 0 (Never) to 2 (Always). Parent and adolescent 
reports on the SPAIC exhibit convergent and construct validity, distinguish adolescents on 
referral status, relate to observers’ ratings of social skills and anxiety, and are sensitive to 


treatment response (Beidel et al., 1995; Beidel et al., 2000a; Beidel, Turner & Morris, 2000b). 
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Parents, adolescents, and unfamiliar peer confederates completed parallel forms of the 
Social Phobia Scale (SPS; Mattick & Clarke, 1998). The SPS is a 20-item measure for assessing 
adolescents’ fears of being scrutinized by others during routine activities (example item: “TI 
become nervous if I have to write in front of other people.”). Informants rated items on a scale 
ranging from 0 (Not at all characteristic or true of me) to 4 (Extremely characteristic or true of 
me). The SPS demonstrates high internal consistency, discriminates individuals on diagnostic 
status, and is sensitive to treatment response (Mattick & Clarke, 1998; Deros et al., 2018). 

In the present study, we used multi-informant SIAS scores in PCA-analyses to identify a 
social anxiety Trait score, and parent and adolescent SPAIC reports and unfamiliar peer 
confederate SPS reports as independent assessments of social anxiety in incremental validity 
analyses. Further, we used parent, adolescent, and unfamiliar peer confederate SPS reports to 
create a composite variable in tests of the incremental value of the Trait score over-and-above 
composite scoring methods. 

Behavioral Tasks 

Task descriptions. Adolescents completed a series of counterbalanced social interaction 
tasks with unfamiliar peer confederates adapted from prior research on social anxiety among 
children, adolescents, and adults (Anderson & Hope, 2009; Beidel et al., 2000a, 2010). 
Participants took approximately 20 minutes to complete these tasks which included a Simulated 
Social Interaction Test (SSIT; adapted from Curran, 1982; Beidel et al., 2000a), Unstructured 
Conversation Task (UCT; adapted from Beidel et al., 2010), and Impromptu Speech Task (IST; 
adapted from Beidel et al., 2010). Detailed descriptions of these tasks and training of unfamiliar 
peer confederates are available elsewhere (Deros et al., 2018; Glenn et al., 2019). Across tasks, 


adolescents interacted with research assistants trained to pose as adolescents. Unfamiliar peer 
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confederates were masked to adolescents’ clinical information, including referral status, and had 
no prior interaction with adolescents. Adolescents’ reactions when interacting with these 
unfamiliar peer confederates predict their reactions when interacting with same-age peers in 
independent tasks (Karp et al., 2018). 

In the SSIT, adolescents participated in a series of five role-playing scenes (e.g., 
offering/accepting assistance, giving/receiving a compliment) with a gender-matched peer 
confederate. In each scene, adolescents had two opportunities to speak and we trained peer 
confederates to provide two scripted responses. In the UCT, adolescents participated in an 
unstructured, adolescent-driven three-minute role-play with a gender-matched peer confederate. 
We provided the following instructions to adolescents: “Pretend that you are at a new school and 
don’t know anyone.” We trained peer confederates to respond neutrally to adolescents and allow 
adolescents to lead the conversation. In the IST, adolescents participated in a speech task in 
which they delivered a speech to a small audience of unfamiliar peers about various topics (e.g., 
public health, politics). Adolescents were provided a three-minute period of preparation after 
which they were provided a ten-minute period to complete their speech. 

Adolescent social anxiety behavioral ratings. We used a behavioral coding scheme 
developed by Beidel and colleagues (Beidel et al., 2000a, 2010; Scharfstein, Beidel, Sims, & 
Finnell, 2011) to obtain independent observers’ ratings of adolescent social anxiety in social 
interaction tasks. Two independent observers used video-recordings of social interaction tasks to 
make macro-level ratings of adolescent social anxiety on a 5-point scale rating from 1 
(Animated) to 5 (Severe anxiety). Independent observers made a total of 7 ratings (i.e., 5 SSIT 
ratings, | UCT rating, 1 IST rating), and we computed a social anxiety composite rating across 


tasks. Inter-rater reliability for the two independent observers’ ratings of social anxiety were in 
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the excellent range (average intraclass correlation = .76). For the current study, we had 105 
participants with complete behavioral rating data available. ! 
Data Analytic Plan 

Trait score identification. We conducted PCA using procedures described by Kraemer et 
al. (2003). Specifically, we conducted an unrotated PCA using three “items”, namely the total 
scores of parent, adolescent, and unfamiliar peer confederate SIAS reports. In this respect, our 
subject-to-item ratio (i.e., 127/3 = 42.3:1) falls well above the typical subject-to-item ratios 
deemed “large” within PCA modeling contexts (e.g., 20:1; see Osborne & Costello, 2004). 
Within this unrotated PCA, we set the number of components to be extracted to three. As 
described previously, we examined principal component weights for each informant’s report to 
determine whether we identified Trait, Context, and Perspective scores. 

Prediction of independent criterion variables. To conduct rigorous tests of the 
incremental validity of an assessment tool and avoid criterion contamination, independent 
criterion variables should be used that are completely independent from individual informants’ 
reports (Garb, 2003). Thus, in our tests of the incremental validity of the Trait score approach, 
we conducted a series of hierarchical linear regressions to evaluate whether the Trait score 
provides incremental prediction of observed adolescent behavior and referral status. We also 
conducted secondary analyses to determine whether the Context and Perspective scores provided 


incremental predication of these independent criterion variables. 


'We conducted chi-square analyses to determine whether participants with and without behavioral rating data 
differed on key demographic characteristics (i.e., adolescent age/gender, family income, parent’s relationship to 
adolescent, parent’s marital status). We did not conduct chi-square analyses for race/ethnicity for these two groups 
given that doing so would require comparisons of groups with cell sizes including fewer than 5 participants. Given 
the exploratory nature of these tests, we applied a Bonferroni correction (i.e., 5 tests and thus a corrected p-value 
of .01). We observed non-significant differences between participants with and without behavior rating data 
available. 
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First, we examined whether the SIAS 7rait score explained variance in observed 
adolescent social anxiety, over-and-above the explanatory value of individual informants’ reports 
on independent social anxiety measures. Behavioral observations contribute to evidence-based 
assessment and clinical decision-making tasks, in particular in assessing behaviors crucial to 
understanding an individual’s presenting problems and measuring change in target behaviors 
over the course of treatment (Groth-Marnat & Wright, 2016). For example, prior work using the 
behavioral coding scheme leveraged in the present study finds that observed behavior can be 
used to estimate treatment response (Beidel et al., 2000b, 2010, 2014; Bunnell, Beidel, & Mesa, 
2013). Thus, in evaluating the Trait score approach, we selected observed behavior as an 
independent criterion variable to determine whether the Trait score could predict social anxiety 
observed in key contexts in which adolescents experience social anxiety. We constructed three 
separate regression models (i.e., one per informant) with observed adolescent social anxiety as 
the criterion variable. For these models, we entered parent (i.e., SPAIC), adolescent (i.e., 
SPAIC), and unfamiliar peer confederate (i.e., SPS) reports of adolescent social anxiety in the 
first step as independent variables, and the 7rait score in the second step as an independent 
variable. These analyses allowed us to determine whether the Trait score provided incremental 
prediction of observed adolescent social anxiety, above-and-beyond prediction provided by well- 
established, commonly used, individual informants’ reports. 

Second, we tested the incremental validity of the SIAS 7rait score using a series of 
hierarchical logistic regressions. We examined whether the 7rait score explained variance in 
adolescent referral status, over-and-above the explanatory value of individual informants’ reports 
on independent social anxiety measures. For clinical decision-making purposes, it is important 


for measurement tools to distinguish youth with clinically significant concerns from those 
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without such concerns (e.g., Achenbach & Rescorla, 2001). In fact, researchers commonly use 
referral status to test the psychometric properties of clinical instruments in research on evidence- 
based assessment of youth anxiety (De Los Reyes & Makol, 2019; Silverman & Ollendick, 
2005). Thus, in evaluating the Trait score approach, we selected referral status as an independent 
criterion variable to determine whether the Trait score approach could be used to distinguish 
youth with and without clinically significant social anxiety concerns. We constructed three 
separate regression models (i.e., one per informant) with adolescent referral status (0 = 
community control, 1 = clinic-referred) as the criterion variable. For these models, we entered 
parent (i.e., SPAIC), adolescent (i.e., SPAIC), and unfamiliar peer confederate (i.e., SPS) reports 
of adolescent social anxiety in the first step as independent variables, and the Trait score in the 
second step as an independent variable. These analyses allowed us to determine whether the 
Trait score provides incremental prediction of adolescent referral status, above-and-beyond 
prediction provided by well-established and commonly used individual informants’ reports. 
Comparing the Trait score and Composite score approaches. We tested whether the 
PCA-derived Trait score provided incremental prediction of our independent criterion variables, 
over-and-above a mean-derived Composite score. We first computed a social anxiety Composite 
score by creating a variable capturing the average of parent, adolescent, and unfamiliar peer 
confederate SPS reports (M = 21.64, SD = 11.28). We selected the SPS to allow for a comparison 
of the Trait score approach to the Composite score approach using independent measures of the 
same construct, each with parallel forms (Trait score: SIAS; Composite score: SPS). We then 
constructed two separate regression models (i.e., one per criterion variable) with either observed 
adolescent social anxiety or adolescent referral status (0 = community control, 1 = clinic- 


referred) as criterion variables. For these models, we entered the SPS Composite variable in the 
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first step as an independent variable, and the SIAS Trait score in the second step as an 
independent variable. These analyses allowed us to determine whether the PCA-derived Trait 
score provided superior prediction when compared to a commonly used integrative method. 

For all of our aims, we interpreted statistical significance using a p-value threshold of <. 
05. We inferred magnitudes of effect sizes based on Cohen’s (1988) effect size conventions for 
the effect size r (low: .10; moderate: .30; large: .50). 

Results 

Preliminary Analyses 

All survey measures displayed acceptable levels of skewness and kurtosis (i.e., scores < 
+2.0) and excellent internal consistency (i.e., all a’s > .93). We computed bivariate correlations 
to examine associations within and between multi-informant social anxiety measures used in our 
study and included this information as supplementary material (Supplemental Table 1). 
Consistent with prior work (Achenbach et al., 1987; De Los Reyes et al., 2015), correlations for 
between-informant ratings of adolescent social anxiety were in the small-to-moderate range (i.e., 
rs = .08-.49). Overall, we observed small-to-moderate correlations among informants’ SIAS 
reports (Figure 2), suggesting that we achieved effective triangulation by selecting informants 
who varied in their context and perspective. 
Trait Score Identification 

In Table 1, we report results from PCA analyses using parent, adolescent, and unfamiliar 
peer confederate SIAS reports. Consistent with Kraemer et al. (2003), we identified three 
components that displayed patterns consistent with Trait, Context, and Perspective scores. 
Specifically, the first component was consistent with a Trait score given that loadings for all 


informants were positive and large in magnitude. The second component was consistent with a 
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Context score given that parent and unfamiliar peer confederate reports exhibited contrasting 
loadings (i.e., reflecting the home vs. nonhome context of informants) and adolescent reports 
loaded between parent and unfamiliar peer confederate reports (i.e., reflecting both contexts). 
The third component was consistent with a Perspective score given that parent and unfamiliar 
peer confederate reports loaded positively and contrasted with adolescent reports (i.e., reflecting 
the other vs. self-perspective of informants). 
Prediction of Observed Adolescent Behavior 

In Table 2, we report analyses examining the incremental validity of the SIAS Trait score 
in predicting observed adolescent social anxiety. In the first step of each regression, individual 
informants’ reports of social anxiety predicted observed adolescent social anxiety at moderate- 
to-large magnitudes (i.e., 8s = .30-.52). In the second step of each regression, the Trait score 
predicted incremental variance in observed adolescent social anxiety, over-and-above the 
variance accounted for by individual informants’ reports (i.e., 6s = .47-.67). Overall, the Trait 
score accounted for an additional 13-29% of the variance in observed social anxiety, representing 
moderate-to-large effects. In additional regression analyses, the Context and Perspective scores 
inconsistently predicted incremental variance in observed adolescent social anxiety, over-and- 
above the variance accounted for by individual informants’ reports (1.e., effects ranging from 
non-significant to large in magnitude; see Supplemental Tables 2 and 3). 
Prediction of Referral Status 

In Table 2, we report analyses examining the incremental validity of the Trait score in 
predicting adolescent referral status. In the first step of each regression, individual informants’ 
reports on independent measures of social anxiety predicted adolescent referral status (i.e., ORs 


= 1.03-1.11). In the second step of each regression, the 7rait score predicted incremental 
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variance in adolescent referral status, over-and-above the variance accounted for by the 
individual informants’ reports. For every one-unit increase in the Trait score in the second step 
of regression models, adolescents were approximately 3 to 7 times more likely to be in the clinic- 
referred group, relative to the community control group. In additional regression analyses, the 
Context and Perspective scores inconsistently predicted incremental variance in adolescent 
referral status, over-and-above the variance accounted for by individual informants’ reports (i.e., 
effects ranging from non-significant to small in magnitude; see Supplemental Tables 2 and 3). 
Comparing the Trait Score and Composite Score Approaches 

We conducted analyses to determine whether the 7rait score provided incremental 
prediction of criterion variables, over-and-above a Composite score calculated by averaging the 
three informants’ SPS reports. We found a large and positive correlation between the Composite 
and Trait score (r = .88, p < .001), and non-significant correlations between the Composite and 
Context (r = -.06, p = .48) and Perspective (r = .004, p = .96) scores. This pattern of correlations 
suggests that although the 7rait and Composite scores show a high degree of overlap, they 
capture unique information derived from informants’ reports. 

Using a hierarchical liner regression model, we examined whether the SIAS Trait score 
provided incremental prediction of observed adolescent social anxiety, over-and-above the SPS 


Composite score. In the first step of the regression model, the Composite score explained 


2 
significant variance in observed adolescent social anxiety, B = .54; AR = .30; FA(1,101) = 
42.55; p < .001. In the second step of the regression model, the Trait score explained incremental 


variance in observed adolescent social anxiety, over-and-above the Composite score, B = .60; 


AR’ = .08; FA(1,100) = 12.75; p <.01. 
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Using a hierarchical logistic regression model, we examined whether the SIAS Trait 
score provided incremental prediction of adolescent referral status, over-and-above the SPS 
Composite score. In the first step of the regression model, the Composite score explained 
significant variance in adolescent referral status, b = .09; OR = 1.10; p < .001. In the second step 
of the regression model, the 7rait score explained significant variance in adolescent referral 
status, over-and-above the Composite score, b = 1.45; OR = 4.26; p < .01. 

To fully evaluate the incremental validity of the Trait score, we conducted additional 
regression analyses in which we examined whether the SPS Composite score provided 
incremental prediction of observed adolescent social anxiety and referral status, over-and-above 


the SIAS Trait score. In the first step of the regression model, the Trait score explained 


2 
significant variance in observed adolescent social anxiety, B = .61; AR = .38; FA(1,101) = 
60.84; p < .001. In the second step of the regression model, the Composite score did not explain 


incremental variance in observed adolescent social anxiety, over-and-above the Trait score, B 


= .01; AR = .00; FA(,100) = .01; p = .94. In the first step of the regression model, the Trait 
score explained significant variance in adolescent referral status, b = 1.30; OR = 3.66; p < .001. 
In the second step of the regression model, the Composite score did not explain significant 
variance in adolescent referral status, over-and-above the Trait score, b = -.02; OR = .99; p =.72. 
Overall, these findings support that the Trait score, which is derived using PCA, provides 
incremental prediction of criterion variables over-and-above well-established individual 
informants’ reports and widely used integrative methods. Specifically, the Trait score, which 
partials out variance from context and perspective (1.e., 7 of “0” between Trait score and Context 
and Perspective scores), provides a more precise index of the domain being assessed, relative to 


both the Composite score and individual informants’ reports. 
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Discussion 

Main Findings 

In this study, we rigorously examined the validity of the Trait score approach when 
applied to adolescent social anxiety assessments. To our knowledge, our study provided the most 
rigorous and conservative test of the 7rait score approach to date, through use of theoretical 
frameworks and basic science to guide selection of informants, parallel measures across 
informants, independent criterion variables, and comparisons to an alternative integrative 
approach. We had three main findings. First, the 7rait score incrementally predicted observed 
social anxiety, over-and-above individual informants’ reports, with moderate-to-large magnitude 
effects. Beyond individual informants’ reports, the Trait score explained approximately 10 to 30 
percent more variance in observed adolescent social anxiety. Second, the Trait score 
incrementally predicted adolescent referral status, over-and-above individual informants’ reports. 
Specifically, for every one-unit increase in the Trait score, adolescents were three to seven times 
more likely to be in the clinic-referred group, relative to the community control group. 
Consistent with Kraemer and colleagues’ (2003) focus on the Trait score, Context and 
Perspective scores failed to predict observed behavior and referral status as robustly and at as 
high magnitudes as the Trait score. Third, we found large magnitude effects for the ability of the 
Trait score to predict observed social anxiety and adolescent referral status, over-and-above a 
mean-derived Composite of informants’ reports of social anxiety. In contrast, the Composite 
score did not provide incremental prediction of observed social anxiety and adolescent referral 
status, over-and-above the Trait score. Overall, these findings provide an initial, rigorous 
demonstration of the Trait score’s validity, supporting the potential of this approach to improve 


clinical decision-making in youth mental health assessments. 
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Research and Theoretical Implications 

Kraemer and colleagues (2003) proposed that the Trait score represents youth behaviors 
that vary across contexts and perspectives, as PCA removes unique variance attributable to these 
components. Researchers commonly apply PCA to individual items in a measure to identify 
subscales or items that appear to measure common constructs (Tabachnick & Fidell, 2001). 
Similarly, Kraemer and colleagues applied PCA to individual informants’ reports to identify 
factors that explain variability among these reports. The 7rait score significantly out-performed 
the Composite score approach in our multi-informant assessment of adolescent social anxiety 
(the reverse was not true), supporting that the PCA-derived Trait score’s “triangulation” may 
provide optimal prediction. This finding suggests that not all integrative strategies are created 
equal. Although the Trait and Composite score were correlated (due to including the same three 
informants’ reports of social anxiety, although on unique measures), the Composite score is 
limited in that it is a simple average of informant’s reports. Using the GPS metaphor, the Trait 
score accounts for the positioning and relative distance of satellites (i.e., informants), whereas 
the Composite score aggregates satellite information without any attention to their source or 
relative distance. More fundamentally, theory underlying the Composite score is in direct 
contrast not only to theory underlying the Trait score but a considerable body of research on 
multi-informant assessments. Specifically, the Composite score approach assumes that each 
informant provides an error-prone estimate that, when aggregated, points to a “true mean” 
representation of the domain assessed (Borsboom, 2005; Edgeworth, 1888). This assumption 
logically extends to collecting reports that achieve maximal collinearity with one another, 
informants for whom reports correspond to a significant extent. In contrast, the Trait score 


approach assumes that informants’ disagreements do not merely reflect measurement error. In 
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fact, they vary systematically and predictably, requiring maximal orthogonality among reports 
(Kraemer et al., 2003). As reviewed previously, research over the last decade supports treating 
informants’ disagreements as reflecting systematic and clinically valuable information (De Los 
Reyes et al., 2013, 2015). Overall, our pattern of findings provides strong support for 
conceptualizing and integrating informants’ reports as orthogonal information sources. 

Over-and-above individual informants’ reports, the Trait score provided incremental 
prediction of observed adolescent behavior and referral status. These criterion variables differed 
in measurement method, providing support for use of the Trait score in prediction of both 
dimensional and discrete clinical indices. As previously mentioned, informants’ reports of youth 
mental health, and particularly youth internalizing problems, are associated with observed 
behavior at only small-to-moderate levels. Benchmarked against prior work, the Trait score 
provided more powerful prediction, accounting for up to 30% more variance in observed 
behavior beyond variance accounted for by individual informants’ reports. Of note, we obtained 
observations of adolescent social anxiety in the context of mock social interactions with 
unfamiliar peers. The 7rait score predicted observed anxiety in the non-home peer context, and 
more strongly than an informant observing adolescent behavior in that same context (1.e., 
unfamiliar peer confederates). Thus, the Trait score outperformed a context-specific informant 
when predicting context-specific behavior, suggesting that it is a useful tool for understanding 
youth mental health concerns globally (i.e., referral status) as well as in specific contexts (i.e., 
interactions with unfamiliar peers). 
Clinical Implications 

In practice, low informant convergence is often interpreted as reflecting informant bias or 


measurement error (De Los Reyes, 2011), resulting in suboptimal use of these reports. 
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Specifically, when confronted with informant discrepancies, clinicians often make decisions that 
“toss out” several informants’ reports in favor of a single informant’s report (Brown-Jacobsen et 
al., 2011; Hawley & Weisz, 2003; Marsh et al., 2019). Taken together with the findings reported 
in this paper, we suspect that one’s hypothesis about the meaning underlying informant 
discrepancies (e.g., they reflect measurement error, one informant provides “better” reports) 
might result in idiosyncratic decision-making for important clinical tasks such as diagnosis and 
treatment selection. A large body of research demonstrates that these idiosyncrasies not only 
commonly occur in clinical work, but that statistical- or actuarial-based decision-making results 
in relative improvements in accuracy and effectiveness (Grove, Zald, Lebow, Snitz, & Nelson, 
2000; Lilienfeld & Lynn, 2014; Meehl, 1954; Youngstrom et al., 2018). Future research is 
needed to understand how the Trait score approach can move clinical decision-making when 
using multi-informant reports from an idiosyncratic, dismissive stance on specific information 
sources, to an evidence-based, inclusive stance on multiple information sources. 

The Trait score approach can be feasibly implemented in clinical settings and is in line 
with current evidence-based assessment practices. Estimation of component scores using PCA 
involves applying a linear transformation to informants’ reports, so as to convert these reports 
into orthogonal variables. Thus, much like calculation of 7-scores in scoring programs for 
commonly used evidence-based assessments (e.g., Achenbach System of Empirically Based 
Assessment [ASEBA]; Achenbach, 1993; Behavioral Assessment System for Children [BASC]; 
Reynolds & Kamphaus, 2004), calculation of the Trait score could be programmed into scoring 
software and included in output designed to enhance clinician interpretation. For example, to 
enhance the clinical utility of the ASEBA measures, Achenbach’s (1993) manual included core 


profiles based on cluster analysis and exposition that detailed a potential empirical taxonomy 
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based on the multivariate analyses. McDermott & Weiss’ (1995) validation study of the 
Adjustment Scales for Children and Adolescents (ASCA) included detailed synopses derived 
from cluster analysis of participants’ 7-score profiles across the ASCA’s six syndromes. Further, 
the technical manual included instructions for determining which cluster best matched a new 
case’s profile; these, too, were programmed into spreadsheets for personal use. More recent 
efforts indicate that one does not need the infrastructure of a well-established clinical instrument 
to apply PCA findings to interpreting clinical data. Specifically, to increase dissemination of 
evidence-based assessment practices in clinical settings, researchers have developed 
straightforward Excel spreadsheets for entering scores for individual clients to obtain 
multivariate profiles (“Focus on Innovation and Redesign in Systems and Treatment”, 2020; 
Youngstrom, 2014). Thus, strategies such as these provide proof-of-concept for the application 
of the Trait score approach in clinical settings. These strategies can inform the development of 
paradigms for using the Trait score as an efficient integrative tool for multi-informant reports 
administered at the individual level and thus commensurate for use in service delivery settings. 
Limitations and Future Directions 

Our study represented an early stage illustration of the validity of the Trait score 
approach. It will be important for future research to determine whether our findings replicate in 
larger samples that vary in key clinical and demographic characteristics. In our study, we 
leveraged the Trait score approach at the sample level, similar to use of factor analytic 
techniques to arrive at subscales for use in refining measurement tools. Although the 
development of strategies for applying the Trait score approach to individual cases awaits further 
study, we expect the current study to inform strategies for use with individual-level clinical 


decision-making. In particular, we encourage future research that leverages both larger and 
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nationally representative samples and norming procedures that allow Trait scores to be 
interpretable at the individual level. 

Kraemer and colleagues (2003) proposed that the Trait score, and not the Context and 
Perspective scores, provides maximal prediction of clinical variables. Consistent with this, our 
findings suggest that the Context and Perspective scores are unlikely to provide robust prediction 
of independent criterion variables. PCA identifies orthogonal principal components, with the first 
principal component accounting for the largest amount of item variance. Thus, it is unsurprising 
that the Context and Perspective scores displayed inferior prediction of independent criterion 
variables as they accounted for less variance in informants’ reports. It is also important to note 
that our findings may be unique given that our study design was optimized for high levels of 
internal validity. That is, by design, we systematically varied informants’ contexts: Parents and 
unfamiliar peer confederates had no access to each other’s observations and as such, we rendered 
impossible opportunities for “cross talk” or the exchanging of clinically relevant information 
between these informants. Despite the likely occurrence of “cross talk” in clinical settings (e.g., 
parent-teacher conferences), a large evidence base supports that informants in these settings 
often provide discrepant reports and patterns among these reports provide meaningful clinical 
information (Achenbach et al., 1987; De Los Reyes et al., 2015). In the Trait score approach, 
informant “cross talk” may be conceptualized as unsystematic measurement error, with 
systematic factors such as informants’ contexts and perspectives accounting for more variance 
among reports. Overall, prior work supports the idea that informant “satellites” can receive some 
overlapping information about youth behavior without undermining effective triangulation. 

We used unfamiliar peer confederates in the Trait score approach given their access to 


adolescents’ social anxiety in peer contexts, where parents often have rare opportunity to observe 
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adolescent behavior (Deros et al., 2018). However, these reports are rarely collected in research 
and clinical settings and may not be feasible to implement in some settings. Future research 
should rigorously examine the Trait score approach among commonly used informants. 
Supporting that the 7rait score approach will generalize to commonly used informants, meta- 
analytic work demonstrates that small-to-moderate convergence is common among parent-youth, 
parent-teacher, and teacher-youth reports of youth mental health (Achenbach et al., 1987; De Los 
Reyes et al., 2015). These levels of convergence approximate those needed to triangulate 
effectively on a Trait score, suggesting that these informants systematically vary in how they rate 
youth mental health. 

Although the present study focused on social anxiety, we expect findings to generalize to 
rigorous research applying the Trait score approach to other problem domains. Future research 
should thoughtfully apply the “mix and match” criterion to identify optimal informants and 
multi-modal indices in mental health assessments. For example, the Trait score approach may 
improve the information derived in externalizing problem assessments. Given the importance of 
understanding externalizing problems across key contexts (i.e., home, school), most research on 
the interpretability of multi-informant reports and cross-contextual variations in mental health 
concerns focuses on externalizing problems (e.g., disruptive behavior; attentional concerns; 
Drabick, Bubier, Chen, Price, & Lanza, 2011; Strickland, Hopkins, & Keenan, 2012). Thus, the 
Trait score approach is well-suited to externalizing problem assessments. 

We also expect that the issues discussed in this paper, which focused on integrating 
multiple information sources, will be relevant to basic research on psychopathology. For 
example, the Research Domain Criteria (RDoC) seeks to develop and validate dimensional 


constructs derived from multiple units of analysis (e.g., subjective reports, behaviors, 


INTEGRATING MULTIPLE INFORMANTS’ REPORTS 33 


physiology), in an effort to improve clinical decision-making (Casey et al., 2014). RDoC 
approaches rely on evidence-based assessment of key constructs as well as data analytic models 
for integrating these multivariate data. Recent work highlights the challenges of accomplishing 
these tasks (Youngstrom et al., 2018). Mental health concerns can be particularly challenging to 
assess, not just due to low convergence among informants’ reports, but also the low convergence 
generally among units of analysis within and across functional domains relevant to externalizing 
disorders (e.g., social processes, regulatory functioning; Beauchaine & Hinshaw, 2020). We 
anticipate that research in this area will benefit from conceptualizing disparate information 
sources as satellites collecting unique information that varies for systematic and predictable 
reasons, and consequently, interpret low measurement convergence as a tool for optimizing 
prediction (De Los Reyes, Drabick, Makol & Jakubovic, 2020). Overall, when integrated with 
contemporary conceptual models informing research on multi-informant assessments (OTM; De 
Los Reyes et al., 2013), the theory and method of the 7rait score approach advanced by Kraemer 
and colleagues (2003) has the potential to transform widely used practices in multi-informant 
assessments, as well as inform clinical developmental research on a broad array of other units of 


analysis. 
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Table 1 


Principal-Component Analysis (PCA) of Parent, Adolescent, and Unfamiliar Peer Confederate Reports on the Social Interaction 
Anxiety Scale (SIAS) (n=125) 


Informant Trait Context Perspective Sources of Variability in Informant’s Report 
Component Weight 
Parent 0:72 0.64 0.26 Home (context), Other (perspective) 
Adolescent 0.82 -0.06 -0.56 Home and Non-home (context), Self (perspective) 
Unfamiliar Peer Confederate 0.75 -0.55 0.37 Non-home (context), Other (perspective) 
Eigenvalue 1.76 0.72 0.52 
Variance attributable to component 58.5% 24.1% 17.4% 


Note. PCA conducted with participants for whom we had full data for parent, adolescent, and unfamiliar peer confederate SIAS 
reports. Kraemer et al. (2003) described the following criteria for each component: Trait (all three informants’ reports load strongly 
and in the same direction), Context (informants from different contexts load in opposite directions) and Perspective (self-reports load 
in the opposite direction of observer informants’ reports). 
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Table 2 
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Hierarchical Regressions Examining the Criterion-Related Validity of SIAS Trait Scores in Predicting Observed Adolescent 


Social Anxiety and Referral Status 


Observed Social Anxiety (V=105) 


Adolescent Referral Status (V=127) 


Variable AR? Total R B(SeB) B b OR 
Step 1 09** 30 
SPAIC, Parent .02 (.01) 20** .11(.02) 1a sea 
Step 2 20e** 62 
SPAIC, Parent -.01(.01) -.10 .07(.03) 1.07"* 
SIAS, Trait Score 55 (.08) 6708" .98(.29) 2.667" 
Variable AR? Total R B(SeB) B b OR 
Step 1 Foe 44 
SPAIC, Adolescent .03 (.01) Agree .08(.02) 1.08*** 
Step 2 QTR 61 
SPAIC, Adolescent -.01(.01) -.06 -.01(.03) 99 
SIAS, Trait Score 54(.10) .66%** 1.35(.36) 3.857 ** 
Variable AR? Total R B(SeB) B b OR 
Step 1 PF fs 52 
SPS, Unfamiliar Peer Confederate .03 (.004) Donte .02(.01) 1.03* 
Step 2 Ager .63 
SPS, Unfamiliar Peer Confederate .01(.01) ae -.04(.02) .96* 
SIAS, Trait Score .39(.09) acre 1.88(.40) 6.53 7*** 


Note. SPAIC = Social Phobia and Anxiety Inventory for Children; SIAS = Social Interaction Anxiety Scale; SPS = Social Phobia 


Scale; *p<.05; **p<.01; ***p<.001. 
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Figure I. Panel A: Example of use of “mix-and-match” criterion to identify optimal informants to include in a multi-informant 
assessment. Informants systematically vary in the perspective and context from which they rate youth mental health symptoms, with 
the goal of effectively triangulating on a Trait score. Panel B: Graphical depiction of multi-informant reports triangulating, much like 
GPS, to identify the Trait score. Both peer- and parent-reports provide information from an other-perspective, with peers providing 
information about the school context and parents providing information about the home context. Youth reports provide the self- 
perspective and information about both the school and home contexts. Figures adapted from Kraemer et al. (2003). 
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Figure 2. Values on dotted lines denote bivariate correlations among parent, adolescent, and unfamiliar peer confederate reports of 
adolescent social anxiety. Values on solid arrows denote component weights from PCA of parent, adolescent, and unfamiliar peer 
confederate reports. *p < .05; **p < .01; ***p <.001. 


49 


